Changes proposed in this pull request

This PR introduces telemetry into Rafiki to observe and measure the growth and activities of the ILP Network. This will provide valuable insights for future improvements and developments. The implementation includes the following key features:

• Rafiki instances are instrumented with Opentelemetry to adhere to a standard format and allow integrating account servicing entities to build their own telemetry.
• Instrumented Rafiki communicates with an Opentelemetry collector through the otel protocol (GRPC).
• Rafiki can send metrics to multiple Opentelemetry collectors, defined in the .env file.
• Telemetry is entirely optional and controlled through an env variable.
• Two counter metrics are introduced: one for counting the number of transactions and another for counting amounts being sent (packet level). These metrics are used to answer three key questions about the network:
  • What is the average amount that a transaction holds?
  • What is the total number of transactions?
  • How much money is being sent through the network?
• A centralized Telemetry service is introduced, which includes a new RatesService. The RatesService reuses the existing Rafiki Rates service but queries a different URL for rates and has a different cache TTL setup. This rates URL is also built on the AWS infrastructure.
• The telemetry data collected by the OTEL collector is further sent to Amazon Managed Service for Prometheus (AMP), and then used in Grafana Cloud to build our "Network insights" dashboards. See the public dashboard here
• Privacy considerations have been a major focus in this PR, with a full documentation on our privacy solution included as an .md file which describes how we implement Local Differential Privacy (LDP).

Infrastructure

The infrastructure for this feature is built on AWS. We are using AWS ECS Fargate, with a cluster of custom ADOT (AWS Distro for OpenTelemetry) collectors. There is also a Network Load Balancer (NLB) in AWS that sits in front of the cluster and load balances the multiple ECS otel collector tasks. All these tasks then send data further downstream to Prometheus. The infrastructure and deployed resource details are documented separately to keep this PR focused on the Rafiki changes.

Context

Full telemetry DOCS

The goal of this PR is to provide a foundation for observing the ILP Network at the money transfer level. By instrumenting Rafiki with Opentelemetry, we can collect data that will help us understand the network's growth and make informed decisions about future developments. The telemetry data is sent to a cluster of Opentelemetry collectors in AWS, but the system is designed to allow integrating ASEs to have their own collectors and build their own telemetry solutions.

Testing

To test out the telemetry feature you can use pnpm localenv:compose up command and everything will be set up for you. If you use the Rafiki peer to peer and cross currency examples in Postman you should see your metrics reflecting on the dashboard after a few minutes.

You can also checkout our tests in packages/backend/src/payment-method/ilp/connector/core/test/middleware/telemetry.test.ts and packages/backend/src/telemetry/meter.test.ts

closes #1913 #1923 #1921 #1956

Checklist

• Related issues linked using fixes #number
• Tests added/updated
• Documentation added
• Make sure that all checks pass
• Postman collection updated